Skip to content

nameparser 2.0 implementation#288

Draft
derek73 wants to merge 218 commits into
masterfrom
v2/core-foundation
Draft

nameparser 2.0 implementation#288
derek73 wants to merge 218 commits into
masterfrom
v2/core-foundation

Conversation

@derek73

@derek73 derek73 commented Jul 12, 2026

Copy link
Copy Markdown
Owner

Implementation branch for the 2.0 design — see the RFC in #285 and the umbrella issue #284 for the design discussion and feedback questions. Implementation learnings that change the design will land as amendment commits on #285, per the plan described there.

⚠️ Merges only at 2.0.0. This branch raises the Python floor to >=3.11 (#257) and drops the typing_extensions conditional dependency (nameparser now has zero runtime dependencies). Master must remain able to cut 1.4.x patch releases for Python 3.10 users, so this PR stays a draft until the 2.0.0 release.

Progress

  • Core data & config modelSpan, Role, Token, Ambiguity/AmbiguityKind, ParsedName; Lexicon, Policy/PolicyPatch/apply_patch, Locale. Frozen, slotted, hashable, picklable dataclasses with eager fail-loud validation; ~110 dedicated tests under tests/v2/.
  • Rendering (render() / initials() / capitalized() / __str__)
  • Parse pipeline + Parser / parse() / parser_for() / matches() + shared case table
  • v1 HumanName facade + CONSTANTS shim (existing test corpus as regression harness) — full v1 suite (1,240+ tests) reconciled and passing against the facade; differential harness (tools/differential/) verifies 1.4-on-PyPI vs the facade over 486 corpus names with one classified diff
  • Locale packs (ru, tr_az) — shipped with the non-interference gate, Provide constants in non-Latin scripts (Cyrillic, Greek, Arabic, Hebrew) #269 default vocabulary, and CLI --locale
  • Documentation rewrite — new-API-first docs (getting started / concepts / customize / locales / migrate + regrouped API reference) and a README that doubles as the PyPI page; every doc code block runs as a Sphinx doctest in CI

Notes on what's here so far

🤖 Generated with Claude Code

derek73 and others added 30 commits July 12, 2026 12:46
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The v2 core design uses enum.StrEnum (3.11+). Full #257 (dropping
typing_extensions, CI matrix) remains tracked on the issue.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Validate that all replace() field values are str (user error if None)
- Append missing-role synthetics in canonical Role order, not kwargs order
- Unquote return type annotation (postponed annotations in effect)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Casefolded seven-component tuple in canonical Role order for dedup,
dict keys, and sorting. Semantic layer comparison; __eq__ remains strict.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds tests/v2/test_layering.py to mechanically enforce the conventions
doc's import layering and the public v2 export surface. Appends the v2
core types (Span, Role, Token, Ambiguity, AmbiguityKind, ParsedName,
Lexicon, Policy, PolicyPatch, PatronymicRule, UNSET, GIVEN_FIRST,
FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST, Locale) to nameparser/__init__.py
alongside the existing v1 HumanName export.

Turns on component-flag mypy strictness for the four v2 modules and
check_untyped_defs for tests/v2, then fixes the resulting test-only
type mismatches by using the precise constructed types (Span(...),
frozenset({...}), tuple-of-pairs) where the literal type didn't matter
to the test, and adding narrow # type: ignore[arg-type] comments only
where a test deliberately exercises runtime coercion/validation of an
intentionally mismatched static type (e.g. Token span/Ambiguity kind
coercion tests, PatronymicRule string coercion, dict aliasing test).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pickle.dumps(Lexicon.default()) raised TypeError because the default
slots-dataclass pickle path serializes the _cap_map MappingProxyType.
Ship every other slot and rebuild the proxy from the canonical
capitalization_exceptions tuple on load. Parser (Plan 3) is picklable
by construction per the core spec, and a Parser holds a Lexicon.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
requires-python is >=3.11 on this branch, so the 3.10 job can no
longer install the package (uv sync either fails or silently
substitutes a managed 3.11). The rest of #257 (typing_extensions,
classifiers) stays tracked on the issue.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With requires-python at >=3.11, Self imports directly from typing and
the conditional dependency can never activate; ruff (UP035/UP036)
flags the dead version block once the floor is raised.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The v1 suite is fully annotated (since #250 put tests/ under mypy), so
pyproject carries no per-file ANN ignores; the new tests/v2 modules
must be annotated too or 'ruff check' fails in CI. Also drops an
unused import ruff flagged (F401).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
derek73 and others added 13 commits July 18, 2026 12:36
Single letter, bare or period-written; of the default suffix words
that means exactly the roman numerals I and V.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'Yamamoto, Haruki' only demonstrates the point if you already know
which word is the surname. 'Thomas, John' is ambiguous to an
English-reading audience in exactly the way the comma resolves.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Append ellipses to the field docs' example parentheticals and teach
the inspection idiom once in the class docstring
(Lexicon.default().<field> is the full list).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each field doc now ends with a same-page link to the config data
constant it sources from (the constants render their complete
contents in the reference's compat section). The renamed pairs
(suffix_words -> SUFFIX_NOT_ACRONYMS, bound_given_names ->
BOUND_FIRST_NAMES) double as visible v1<->v2 mapping documentation,
and particles_ambiguous -- which has no constant -- documents its
derivation from NON_FIRST_NAME_PREFIXES instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same enum-member invisibility as AmbiguityKind: without doc comments
autodoc skipped them, so the class docstring promised seven fields
the page never listed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The fold example (verified live): 'Gabriel García Márquez' -> family
'García Márquez' for multi-part-surname data.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review feedback (Derek): 'How the parser thinks' implied
nondeterminism -> 'How the parser works'; the 'four short essays'
opener and 'pipeline, in one breath' phrasing replaced with direct
statements; 'From string to name' now walks the canonical example
concretely (8 tokens, Dr. span (0,3), family as a view joining
de/la/Vega) and defines span before explaining what it prevents; the
three-container bullets lead with the container names; the Dean
paragraph returns to the v1 docs' problem-first framing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DEFAULT_NICKNAME_DELIMITERS grows by the eight conventions the issue
proposed: smart quotes, low-high (de/pl/cs/hu), right-right (sv/fi),
guillemets both directions (« Petit » inner spacing tolerated), CJK
corner brackets, fullwidth parentheses. Curly single quotes stay
excluded (U+2019 is the apostrophe in O'Connor, pinned). Both APIs
get them: the shim's sentinel map gains named keys
(smart_double_quotes, guillemets, ...) so the v1 keyed idioms
(pop/move/del) work on the new pairs like the original trio.

The conventions share characters in opposite roles ('"' closes the
German pair but opens the English one; '»' closes guillemets and
opens the reversed pair), which made extraction emit spurious
unbalanced-delimiter ambiguities: extract now tracks unmatched-open
offsets and drops candidates inside a region another pair
successfully masked. Genuine unbalanced opens still flag.

TDD throughout (extract suppression -> defaults -> shim sentinels,
each red first); 9 case rows through both runners; differential
harness exit 0 unchanged (no typographic quotes in the corpus);
release-log feat entry; reference/customize/migrate docs updated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
BOUND_FIRST_NAMES had only the transliterations; the script originals
now join too: عبد (one entry covers abdul/abdel/abdal -- script
attaches the article to the following word), the kunya pair أبو/ابو,
and أم/ام. Behavior mirrors the Latin twins, probed live before
pinning: bound join fires with 3+ tokens (عبد الرحمن محمد -> given
'عبد الرحمن'), the two-token kunya stays split (أبو مازن unchanged),
and non-leading أبو still prefix-chains onto family (أحمد أبو خليل
unchanged -- both prior pins hold). TDD: 4 case rows red in both
runners first. Differential exit 0 unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Twelve given-name titles -- Arabic honorifics precede the given name,
so all join الشيخ in FIRST_NAME_TITLES: the article forms (never
given names) الدكتور/الدكتورة/الأستاذ/الأستاذة/الحاج/الحاجة/الشيخة
and the bare doctor/professor/engineer forms
دكتور/دكتورة/أستاذ/أستاذة/مهندس. Deferred under the collision rule
(recorded in the data-module comment): bare سيد/شيخ/أمير/سلطان (all
common given names), the د. abbreviation (bare د would swallow
single-letter initials, the bare-κ trap), and the Ottoman
post-nominals باشا/بك/أفندي (survive as family names).

و ('and') joins conjunctions: formal script attaches it to the next
word, but the standalone informal spacing is common in real data;
single-character like y/и, so the #11 carve-out protects short names.

TDD: 20 rows red first, incl. the الحاج + عبد الرحمن compound
(title + bound join + family) and the 6-piece و title chain.
Differential exit 0 unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
derek73 and others added 16 commits July 18, 2026 14:11
…ow-up)

Hebrew titles גברת/פרופ'/פרופ׳/פרופסור/עו"ד/עו״ד/הרב join מר as
plain titles (Israeli convention: family follows), each abbreviation
in both geresh/gershayim spellings per the ד"ר precedent. Hebrew
gains the suffix category Arabic deferred: ז"ל and שליט"א (both
spellings) -- post-nominals ubiquitous in genealogical data, never
names, mid-word quotes inert in extraction.

Devanagari becomes a #269 script: श्री/श्रीमती/डॉ (डॉ. matches via
edge-period normalization). Latin sri/shri deliberately NOT added --
the transliterations collide with real given names (Sri Mulyani);
the native script cannot.

Deferred under the collision rule, recorded in the data-module
comments: bare רב (the ordinary word 'many') and בר as a particle
(Bar is a common modern Israeli given name; the surname spelling is
hyphenated anyway).

TDD: 15 rows red first across title/suffix/script categories.
Differential exit 0 unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three review agents independently found the same defect in the #273
defaults: each pair scanned the full text independently in sorted
order, so when two conventions sharing a character in opposite roles
BOTH occurred ('Hans „Erster" und "Zweiter" Müller'), the
earlier-sorted pair claimed the other's close as its own open,
extracted a bogus span across the real boundary, and the legitimate
match was dropped by the overlap check with zero ambiguity signal.

extract_delimited now runs one position-driven scan: at each position
the leftmost boundary-valid opener among ALL pairs wins, with
per-pair forward-only cursors keeping the scan linear. Overlapping
matches become impossible by construction, so the overlap check on
matches is gone; the unmatched-open filter remains (now via
_overlaps) for bulk-recorded dangling opens consumed by later
matches. Bucket precedence is documented where it actually lives:
Policy's maiden-wins canonicalization (facade: shim pre-subtraction)
-- the stale 'maiden scanned first' module-docstring claim and the
contradicting 'nickname first' comment are both gone.

Also the review's quantified perf fix: a cached delimiter-charset
prescreen short-circuits the no-delimiter common case (isdisjoint),
more than paying back the 3->11 default-pair growth.

TDD: four mixed-convention/coexistence tests red first; 19 extract
tests green; differential exit 0 unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- name_order's comma claim narrowed: only a family-separating comma
  ignores it; 'John Smith, Jr.' still obeys name_order (live-proven,
  SUFFIX_COMMA reads it).
- The 'issue #11' carve-out citations were pointing at GitHub #11 (a
  2014 'meant to create a milestone' accident); the real source is
  GOOGLE CODE issue 11, the 'john e smith' bug -- v1's own comment
  carries the full URL. All six sites now say so.
- _DelimiterManager no longer claims 'only the three named sentinels'.
- The בר particle deferral moved to prefixes.py with the other
  particle decisions.
- The patronymic migration hint is a single module constant shared by
  Policy and PolicyPatch.
- apply_patch documents maiden-wins through patches as intended
  (decided 2026-07-19): maiden membership IS the routing decision,
  whoever contributes it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Match-time contract of lower()-not-casefold() pinned at parse level
(ΚΟΣ and GROẞFÜRST match; ASCII-SS GROSSFÜRST deliberately does not,
v1 parity); facade rows for each new vocabulary category (bound
given, Hebrew suffix, Devanagari title); maiden-wins pinned on a
typographic pair; a pre-#273 three-key pickle restores as exactly its
own keys (setstate replaces, never merges defaults).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
"core spec §5a/§5b/§6" pointed at docs/superpowers/, which is
gitignored and not committed on any branch -- a future reader can
never resolve the citation. Replace each with the actual rule content
inline instead (review finding on #288).
patronymic_rules already probed with iter() to give non-iterable
values (an int, a bool) a curated, field-naming TypeError. The other
iterable-typed fields (name_order, nickname_delimiters,
maiden_delimiters, extra_suffix_delimiters, every Lexicon vocab field,
capitalization_exceptions) fell through to whatever bare "'int' object
is not iterable" the first tuple()/iter() call happened to raise.
Apply the same _require_iterable probe everywhere, with regression
tests (review finding on #288).
role = Role.GIVEN when a main-stream token reaches assemble() with no
role set was previously untested -- assign/group always set a role in
the real pipeline, so nothing exercised this last-resort branch.
Construct the invariant violation directly (an unassigned-role
WorkToken) so a future refactor can't silently change or lose this
documented safety-net behavior (review finding on #288).
The layering rule forbids the pipeline/_render from importing
nameparser.config directly, so several patterns are copied by hand
with "keep in sync by hand" comments -- nothing previously checked
that promise. Tests may legally import both sides (test_layering.py's
own convention), so add sync tests for every duplicated
pattern/table, including the documented "initial" trailing-"?"
deviation between config and the pipeline copies (review finding on
#288).
Several branches were only reached through tests/v2/test_cases.py's
full-pipeline corpus, so a regression scoped to one stage wouldn't be
pinpointed by that stage's own test file: assign's SUFFIX_COMMA
structure, segment's cosmetic-vs-structural empty-bucket handling
(single trailing comma vs. a leading comma), group's
extra_suffix_delimiters entry-splitting in a FAMILY_COMMA tail
segment, and tokenize's interleaving of two simultaneously active
extracted regions (review finding on #288).
"always a subset" read as identity to a casual reader, but the
constructor check uses `in` (Token's == equality): two distinct Token
instances with identical text/span/role/tags satisfy it. Document the
distinction at the field, the class docstring, and the check site
(polish item from the #288 review).
name_order and the union (set-valued) fields get an iter()/type probe
in __post_init__; the four scalar override fields (middle_as_family,
lenient_comma_suffixes, strip_emoji, strip_bidi) deliberately get none
until apply_patch runs -- previously true only per the class docstring
and the tests, with nothing at the loop itself flagging the omission
as intentional rather than a gap (polish item from the #288 review).
The prose in each comment already names the specific v1 logic being
ported; the hardcoded line numbers (1285, 1313, 1333, 1368, 1390) add
nothing on top of that and would silently go stale on the next edit to
parser.py, with nothing to catch the drift (polish item from the #288
review).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Retitle "three containers" section and swap the Policy example from
  name_order (which actually correlates with language for Chinese/
  Hungarian/Vietnamese names) to nickname_delimiters, which doesn't.
- Note that capitalization_exceptions keys match punctuation-normalized
  text, so "phd" also covers "Ph.D.".
- Fix a stale nickname_delimiters example (guillemets are already in
  the default set) and drop an unneeded frozenset() wrapper.
- Replace the rendering-arguments stub signatures with links to the
  real methods plus a worked, doctested example, including
  capitalized(force=True) for genuinely mixed-case input.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Derek's terminal find: str(parse('Von Johnson (smith)')) silently
dropped the nickname. v1's default string_format always rendered
'({nickname})'; the 2.0 default had lost it. The default spec is now
'{title} {given} {middle} {family} {suffix} ({nickname}) née
{maiden}' -- v1's nickname position restored, maiden appended with a
marker that round-trips (parens re-extract as nickname, née is a
maiden marker, pinned by a reparse assertion). The #254 collapse
learns to drop the trailing orphaned 'née' (and the bare-née
all-empty case); content 'Née' is untouched (case-exact match). The
HumanName facade keeps v1's own string_format default.

TDD: red first; differential exit 0 (facade rendering unchanged).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hesized maiden

'{title} {given} "{nickname}" {middle} {family} ({maiden}) {suffix}':
nickname in the classic quoted position after the given name, maiden
parenthesized after the family. The collapse's existing empty-'""'
and empty-'()' rules cover both decorations, so the née trailing-strip
special case is deleted. Trade-off documented: the quoted nickname
round-trips exactly; parenthesized maiden re-parses as a nickname
(custom 'née {maiden}' spec available for lossless round-trip).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants